Skip to content

Preserve schema metadata when annotating Arrow fields - #295

Merged
timkpaine merged 2 commits into
Point72:mainfrom
pengpengyi92:fix/arrow-field-metadata-preserve-schema-20261112
Nov 12, 2026
Merged

timkpaine merged 2 commits into
Point72:mainfrom
pengpengyi92:fix/arrow-field-metadata-preserve-schema-20261112

Conversation

@pengpengyi92

Copy link
Copy Markdown
Contributor

Description

add_field_metadata() rebuilds the Arrow schema with new field metadata, but
pa.schema(new_schema) silently discards the original schema-level metadata.
This happens even when the supplied mapping is empty or only names an absent
column.

In particular, Table.from_pandas() stores index reconstruction information
under the schema's pandas key. Annotating a column currently resets a named
RangeIndex, or turns named Index/MultiIndex fields into ordinary data columns
on to_pandas().

Retain table.schema.metadata when constructing the replacement schema. The
existing selected-field replacement semantics, untouched fields, data types,
nullability, and values remain unchanged. No dependency or API changes.

The tests cover raw/binary table metadata, empty/unmatched mappings, named
RangeIndex/Index/MultiIndex round trips, and the existing selected-field
replacement behavior with no schema metadata.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactor / code cleanup
  • CI / build configuration

Validation

Windows, Python 3.12.13, PyArrow 26.0.1, pandas 3.1.1, Polars 2.1.0.

python -m pytest ccflow/tests/utils/test_arrow.py ccflow/tests/test_arrow.py \
  ccflow/tests/exttypes/test_arrow.py ccflow/tests/result/test_pyarrow.py \
  -q -o addopts= -p no:cacheprovider -p no:asyncio
33 passed, 6 subtests passed

With the new regression tests and the original implementation, all six
metadata/index subtests fail; the other seven test cases pass. With the fix,
the utility module reports seven passed and six subtests passed.

The broader local selection (all utils plus the Arrow and file-publisher
tests) is not fully green on this Windows host: 319 passed, 24 failed.
The clean upstream commit 05ce2450aaa53d865bca9cb68903900b039fa032 has
316 passed and the exact same 24 failing test IDs under the same interpreter
and dependencies. Failures include Windows/Hydra path assumptions, CLI tests,
and temporary-directory access errors; this PR does not change those areas.

Checklist

  • Changed Python files pass Ruff lint and formatting checks
  • Arrow-focused tests pass; broader Windows baseline limitations disclosed above
  • Regression tests added
  • Function docstring updated
  • DCO sign-off included with the contributor's explicit authorization
  • Full make lint / make test CI matrix (not claimed locally)

Out of scope: metadata handling in type-conversion helpers, metadata merging,
recursive conversion, storage adapters, and unrelated Windows test fixes.

Signed-off-by: Pengyi Peng <pengpengyi92@users.noreply.github.com>
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.85%. Comparing base (402a0ad) to head (fc62137).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #295   +/-   ##
=======================================
  Coverage   93.84%   93.85%           
=======================================
  Files         205      205           
  Lines       22920    22944   +24     
  Branches     1509     1511    +2     
=======================================
+ Hits        21509    21533   +24     
  Misses       1120     1120           
  Partials      291      291           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timkpaine
timkpaine merged commit 83f2b0a into Point72:main Nov 12, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants